Search for: All records

This work-in-progress research paper describes a study of different categorical data coding procedures for machine learning(ML) in engineering education. Often left out of methodology sections, preprocessing steps in data analysis can have important ramifications on project outcomes. In this study, we applied three different coding schemes (i.e., scalar conversion, one-hot encoding, and binary) for the categorical variable of Race across three different ML models (i.e., Neural Network, Random Forest, and Naive Bayes classifiers) looking at the four standard measures of ML classification models (i.e., accuracy, precision, recall, and F1-score). Results showed that, in general, the coding scheme did not affect predictive outcomes as much as ML model type did. However, one-hot encoding – the strategy of transforming a categorical variable with k possible values to k binary nodes, a common practice in educational research – does not work well with a Naive Bayes classifier model. Our results indicate that such sensitivity studies at the beginning of ML modeling projects are necessary. Future work includes performing a full range of sensitivity studies on our complete, grant-funded project dataset that has been collected, and publishing our findings. 

STEM undergraduate instructors teaching remote courses often use traditional lecture-based instruction, despite evidence that active learning methods improve student engagement and learning outcomes. One simple way to use active learning online is to incorporate exploratory learning. In exploratory learning, students explore a novel activity (e.g., problem solving) before a lecture on the underlying concepts and procedures. This method has been shown to improve learning outcomes during in-person courses, without requiring the entire course to be restructured. The current study examined whether the benefits of exploratory learning extend to a remote undergraduate physics lesson, taught synchronously online. Undergraduate physics students (N = 78) completed a physics problem-solving activity either before instruction (explore-first condition) or after (instruct-first condition). Students then completed a learning assessment of the problem-solving procedures and underlying concepts. Despite lower accuracy on the learning activity, students in the explore-first condition demonstrated better understanding on the assessment, compared to students in the instruct-first condition. This finding suggests that exploratory learning can serve as productive failure in online courses, challenging students but improving learning, compared to the more widely-used lecture-then-practice method. 

This WIP paper presents new research on exploratory learning, an educational technique that reverses the order of standard lecture-based instruction techniques. In exploratory learning, students are presented with a novel activity first, followed by instruction. Exploratory learning has been observed to benefit student learning in foundational math and science courses such as calculus, physics, and statistics; however, it has yet to be applied to engineering topics such as programming. In two studies, we tested the effectiveness of exploratory learning in the programming unit of a first-year undergraduate engineering course. We designed a new activity to help students learn about different python error types, ensuring that it would be suitable for exploration. Then we implemented two different orders (the traditional instruct-first versus exploratory learning’s explore-first) across the six sections of the course. In Study 1 (N=406), we did not detect a difference between the instruct-first and explore-first conditions. In Study 2 (N=411), we added more scaffolding to the activity. Students who received the traditional order of instruction followed by the activity scored significantly higher on the assessment. These findings contradict the exploratory learning benefits typically shown, shedding light on potential boundary conditions to this effect. 

This research work in progress research paper examines student perceptions after completing an exploratory learning lesson before instruction on an introductory programming concept. During exploratory learning activities, students explore a novel concept prior to instruction—the reverse of typical instruct-then-practice methods. Exploratory learning before instruction can help students activate prior knowledge, become aware of their knowledge gaps, and discern important problem features to improve conceptual understanding. Students in a first-year engineering course (N=402) learned about Python error messages in one of two conditions. In the explore-first condition, students completed a collaborative activity prior to instruction. In the instruct-first condition, students received instruction prior to the activity. Following the activity and instruction, students completed a survey to assess their perceptions of the activities. Survey items (e.g. cognitive load, self-efficacy, belonging, knowledge gaps) were chosen as potential factors that could explain learning outcomes between the two conditions. In prior work, we found higher posttest scores in the instruct-first compared to explore-first condition, contrary to the majority of previous studies. Cognitive load and knowledge gaps were higher in the explore-first condition than the instruct-first condition. Self-efficacy and competence were lower in the explore-first condition. No other significant differences were found. Exploring before instruction might disrupt learning and perceived efficacy and competence if the activity is too challenging, or if the instruction does not fully resolve gaps in students’ knowledge. 

The focus of our current National Science Foundation (NSF: IUSE Award #1912253) project is the degree to which spaced retrieval practice, as compared to massed, increases performance on an end-of-semester exam in 10 different STEM courses (e.g., engineering, chemistry, biology). This paper presents an intermediate analysis examining the psychometric properties of the retrieval-practice exercises in two specific courses. The critical question is whether the psychometric properties of the exercises differ depending on whether they are spaced or massed. Preliminary results indicate that spacing does not consistently affect reliability of items, but can impact item difficulty. 

This study examined the difficulty introduced by spaced retrieval practice in Calculus I for undergraduate engineering students. Spaced retrieval practice is an instructional technique in which students engage in multiple recall exercises on the same topic with intermittent temporal delays in between. Spacing out retrieval practice increases the difficulty of the exercises, reducing student performance on them. However, empirical research indicates that spaced retrieval practice is associated with improvements in students’ long-term memory for the retrieved information. The short-term costs and long-term benefits of spaced retrieval practice is an example of desirable difficulty, when more difficult exercises during the early stages of learning result in longer-lasting memory [1]. With support from the National Science Foundation (NSF), we sought to address: Does spacing decrease performance on retrieval practice exercises in an engineering mathematics course? Results showed that student performance was significantly lower for questions in the spaced condition than questions in the massed condition, indicating that we successfully increased the difficulty of the questions by spacing them out over time. Future work will assess final quiz performance to determine whether spacing improved long-term course performance, i.e., whether the difficulty imposed by spacing was desirable. 

This Full-Length Research Paper investigates the difficulty imposed by spaced retrieval practice in nine introductory Science, Technology, Engineering, and Mathematics (STEM) courses. By improving student performance in these courses, evidence-based pedagogical practices have the potential to increase graduation and success in STEM fields. Spaced retrieval practice is a technique in which questions on the same topic are asked repeatedly over time with intermittent delays. Spacing may initially make retrieval more difficult because it requires learners to recall information from long-term as opposed to short-term memory. However, this difficulty may ultimately be “desirable” because spacing often produces memory benefits in the long-term. The current paper examines the difficulty imposed by spaced retrieval in the nine STEM courses, using data collected from a 3-year project funded by the National Science Foundation. Results indicated that the magnitude of the difficulty imposed by spacing varied widely across the diverse STEM barrier courses. We anticipate that we will find similarly wide variability in the effectiveness of spaced retrieval practice in students' final learning outcomes, which will be investigated in future work. 

This WIP Research Paper investigates the temporal nature of the difficulty imposed by spacing. Spaced retrieval practice is an evidence-based strategy for improving memory and consists of asking multiple questions on a topic with intermittent delays. Spacing is often thought to impose difficulty by making questions harder to answer. However, this difficulty may be desirable, since spacing ultimately improves memory. In this paper, we (a) outline an implementation of spaced retrieval practice in an engineering mathematics classroom, (b) describe the development of an SQL database to organize and manage the large and complex dataset, and (c) discuss a brief but interesting dive into the rich data we have collected. Statistical analyses revealed that, when three questions targeting the same topic are spaced over multiple quizzes, versus being massed on a single quiz, students are less likely to answer the first and second questions correctly. Spacing does not affect students' ability to answer the third questions. This suggests that spacing may impose difficulty when students are first learning to perform mathematical operations, rather than when they are trying to retrieve memories of how to perform those operations.